\chapter{Performance Analysis}

There are a few general performance issues that we were concerned about the LAMS prototype. The following section outlines performance analysis methods we used to answer and investigate the following issues:

\begin{itemize}
\item How many VoIP servers can LAMS handle?
\item How much traffic is contributed by LAMS monitoring one to many VoIP servers?
\item How many VoIP clients can our VoIP server handle?
\item Is the performance of the prototype system adequate for our scenario?
\item Is the visual design of the LAMS virtual world, and our choices for the visual characteristics of entities interpreted correctly by new users of the LAMS system?
\end{itemize}

\section{CPU \& Memory Requirements of IP Telephony}

VoIP servers are meant to be robust and should handle millions of calls every day. The majority of the time the maximum call limit will never be reached but it is possible. Provisioning and trunking theory is a topic telecommunications companies have to deal with \cite{wirelessbook} \cite{www:trunkingtheory}. Over-provisioning results in a waste of bandwidth while under-provisioning results in poor server. A balance has to be found.

To test how many simultaneous channels and calls our system could handle we performed maximum channel testing and call maximum VoIP performance tests. Our testing method was guided by \cite{www:sippPerformance} and \cite{www:sippPerformWiki}. We used Asterisk PBX software as our VoIP server running on a standard install of FreeBSD 7.0 with an Intel Celeron 2.8 GHz with 1024 MB of RAM. After each test, the VoIP server was restarted to give consistent results. Asterisk and a memory/cpu logger were the only significant processes running on the machine.

We generated SIP phone calls using a SIP generator developed by Hewlett Packard called SIPp \cite{www:sipp} . SIPp can set up calls and generate many SIP scenarios. We used it to simulate a User Agent Client (UAC) or User Agent Server (UAS) scenario with real RTP data. A maximum call limit can be placed which SIPp will always try to reach. 

We performed the following call scenarios in table \ref{tab:sipcalls} for each test.


\begin{table}[h]
\centering
\begin{tabular}{| l |}
\hline
1 SIP call\\ \hline
5 SIP calls\\ \hline
10 SIP calls\\ \hline
20 SIP calls\\ \hline
50 SIP calls\\ \hline
100 SIP calls\\ \hline
200 SIP calls\\ \hline
300 SIP calls\\ \hline
\end{tabular}
\caption{Simultaneous SIP call tests performed}
\label{tab:sipcalls}
\end{table} 

\subsection{Maximum Voice Channels }

To perform maximum channel testing, a SIP client dials a number hosted by the VoIP server which plays back a media file. Only one channel is used by the client and the VoIP server is forced to process the media file to be played back. 

We use the setup in Figure \ref{fig:echoTesting} where SIPp is used to initiate a SIP call/s to the VoIP system. Each call only lasts 20 seconds, but SIPp would always try to keep the call rate at the specified value. 

\begin{figure}[htp]
\centering
\includegraphics{diagrams/echoTesting.pdf}
\caption{Maximum Channel Testing Setup}
\label{fig:echoTesting}
\end{figure}

% following? which tests?


Using the test scenarios outlined in table \ref{tab:sipcalls}, we monitored both the CPU usage and memory allocation of the Asterisk process. The results are shown in Figures \ref{fig:echoTestingCPU} and \ref{fig:echoTestingMEM}. 

\begin{figure}[htp]
\centering
\includegraphics{diagrams/echoTestingCPU.pdf}
\caption{CPU usage of Asterisk as the calls are increased.}
\label{fig:echoTestingCPU}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics{diagrams/echoTestingMEM.pdf}
\caption{Memory usage of Asterisk as the calls are increased.}
\label{fig:echoTestingMEM}
\end{figure}

Our findings show that as the number of calls increased so did the CPU and memory usage. Asterisk used a maximum of 8\% CPU time (Figure \ref{fig:echoTestingCPU}) and was allocated by the operating system 100 MBs of RAM (Figure \ref{fig:echoTestingMEM}) during the period when 300 simultaneous call were being attempted.

During our testing we reached a SIP call channel limit. Asterisk would not create more than 217 channels. This may have been due to either default configuration of Asterisk or FreeBSD limitation on allowing a process to use a certain amount of CPU.
\FloatBarrier

\subsection{Call Maximum}
\label{sec:callmax}
A typical VoIP call consists of two clients and a VoIP server in between them \cite{www:understandingSip}. The VoIP server requires to process the incoming call and forward it to the correct location. This time two channels are created, one for the caller and the other for the recipient. Using SIPp simultaneous call capabilities, the same call tests (table \ref{tab:sipcalls}) were conducted. Real RTP packets were used from UAC to UAS. Figure \ref{fig:callMaxtesting} describes our setup followed by our results in Figure \ref{fig:sippUACUASCPU} and FIgure \ref{fig:sippUACUASmem}. 

\begin{figure}[htp]
\centering
\includegraphics{diagrams/sippCallMaximum.pdf}
\caption{Client Setup}
\label{fig:callMaxtesting}
\end{figure}


\begin{figure}[htp]
\centering
\includegraphics{diagrams/sippUACUASCPU.pdf}
\caption{CPU usage of Asterisk as calls are increased}
\label{fig:sippUACUASCPU}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics{diagrams/sippUACUASmem.pdf}
\caption{Memory usage of Asterisk as the calls are increased.}
\label{fig:sippUACUASmem}
\end{figure}
\vspace{4in}
Results conclude the CPU usage was overall higher compared to channel maximum testing (Section \ref{sec:callmax}). This is probably due to SIP calls having to be processed and then the VoIP server having to create another channel for the recipient of the call. Maintaining the channel would also contribute to the CPU usage. Overall the memory consumption is far less when compared to echo testing. A plausible reason is that Asterisk did not have to play back a recorded message compared to the maximum channel testing. 

Again we reached a limit as maximum simultaneous calls reached by SIPp was around 95 - 105 during our testing. Despite the limit we continued with the rest of the tests outlined in table \ref{tab:sipcalls}. Each call requires two channels set up by the VoIP server which correlates the channel limit of 217 in section \ref{sec:callmax}. 

Our findings show that our default VoIP server setup using Asterisk PBX can handle between 95 - 105 calls. To be safe we state that no more than 90 calls should be processed by our VoIP server. 

\section{Network Performance of LAMS}

\subsection{Impact of Collecting Server and Telephony Statistics}
LAMS requires all VoIP servers that are being monitored to periodically send system state updates. One of the goals of LAMS is for the network traffic it produces not to impede on the network. To test the network traffic produced from the VoIP server to LAMS server, we performed traffic analysis for three scenarios.

\begin{table}[h]
\centering
\begin{tabular}{| l |}
\hline
Traffic generated by 1 VoIP Server to LAMS \\ \hline
Traffic generated by 2 VoIP Servers to LAMS \\ \hline
Traffic generated by 3 VoIP Server to LAMS \\ \hline
\end{tabular}
\caption{Scenarios for our traffic analysis of monitoring multiple VoIP servers}
\label{tab:voipserverscenarios}
\end{table} 

Each VoIP server underwent identical SIP generation techniques to give consistent network traffic. The LAMS server and VoIP servers were connected by a switch over 100Mbps Ethernet LAN with no other hosts on the network to interfere. \textit{tcpdump} was used on the LAMS server to capture update traffic (Ganglia UDP packets) coming from each VoIP servers. The following Figures (\ref{fig:gangliaTraff1pbx} \ref{fig:gangliaTraff2pbx} \ref{fig:gangliaTraff3pbx}) demonstrate the traffic produced as each VoIP server is added to the monitoring list.

\begin{figure}[htp]
\centering
\includegraphics{diagrams/gangliaTraff1pbx.pdf}
\caption{Characteristic of state update traffic of from one VoIP servers to LAMS server.}
\label{fig:gangliaTraff1pbx}
\end{figure}

As can be seen from Figure \ref{fig:gangliaTraff1pbx} , traffic from just one VoIP server is bursty but negligible compared to the traffic of a small ISP  \cite{www:smallISPtraffic}. A burst 45 - 52 packets (6264 bytes)  is sent every 18 seconds while at all other times it hovers at 110 bytes.  This coincides with code written to extract telephony statistics from Asterisk PBX and send it to LAMS. The overall capture had a average of 426.83 bytes per second.

\begin{figure}[htp]
\centering
\includegraphics{diagrams/gangliaTraff2pbx.pdf}
\caption{Characteristic of state update traffic of from two VoIP servers to LAMS server.}
\label{fig:gangliaTraff2pbx}
\end{figure}

\vspace{4in}
Adding a second VoIP server (Figure \ref{fig:gangliaTraff2pbx}) increases the number of traffic bursts of 6264 bytes. It is clear that there are double the amount of spikes. The overall capture had an average of 979.5 bytes per second. 

\begin{figure}[htp]
\centering
\includegraphics{diagrams/gangliaTraff3pbx.pdf}
\caption{Characteristic of state update traffic of from three VoIP servers to LAMS server.}
\label{fig:gangliaTraff3pbx}
\end{figure}

The third VoIP server (Figure \ref{fig:gangliaTraff3pbx}) increases the spikes and also increases the probability of one update bursts occurring in phase with another burst from another VoIP server. The overall capture had an average of 1485.89 bytes per second. 
\vspace{2in}

Using the obtained traffic rates from our traffic captures and linear regression, we predict the following traffic patterns for each additional VoIP server. 


\begin{figure}[htp]
\centering
\includegraphics{diagrams/gangliaProjectedTraffic.pdf}
\caption{Projected VoIP server update traffic to LAMS server.}
\label{fig:gangliaProjectedTraffic}
\end{figure}


Figure \ref{fig:gangliaProjectedTraffic} suggests that LAMS can monitor 20 VoIP servers being monitored at once and have an aggregated 10 Kbytes a second of network traffic affect. According to a small ISP \cite{www:smallISPtraffic} this is negligible. The theoretical maximum all depends on the conditions of a real network LAMS is placed in. 

From our results it is clear to see that if a VoIP system state changes within 18 second spikes (eg a 15 second phone call), then the monitoring tool (Appendix \ref{ganglia}) developed to extract telephony statistics is far too slow. A quicker implementation must be developed if LAMS were to be deployed commercially to VoIP service providers. 


\subsection{Impact of Updating the LAMS Virtual World}
Periodically the Grazer input daemon sends updates to LAMS, instructing entities to change their appearance and display information that reflects the current state of the VoIP system. The frequency of these updates is configurable and ideally should be the same frequency at which statistics are collected from the VoIP servers. As with collecting telephony statistics a trade off exists between the responsiveness of the virtual world and the network load created by updating LAMS.

To test the network load of Grazer, the Grazer input daemon was run on Host A while the LAMS server was run on Host B. The two hosts were connected by a switch over 100Mbps Ethernet LAN, the latency between the hosts was negligible. Tcpdump on Host B was used to capture the traffic that Grazer sent to LAMS during a five minute interval.

It was found that during each update period Grazer generated 22.6Kbytes of UDP packets to LAMS, and then was idle while waiting for the next update period. Each update generated the same amount of traffic. This traffic pattern in shown in Figure \ref{fig:grazerTraff}, where the update interval was 30 seconds.

\begin{figure}[h]
\centering
\includegraphics[width=5in]{diagrams/grazerTraff.pdf}
\caption{Characteristic Traffic of The Grazer Input Daemon. Time intervals of 10ms.}
\label{fig:grazerTraff}
\end{figure}

By inspecting a single update period closely, as shown in Figure \ref{fig:grazerTraffClose}, it can be seen that the data rate was fairly constant during an update. For this 100Mbps Ethernet scenario an update took 460ms to complete and during this time generated 403Kbps of traffic.

\begin{figure}[h]
\centering
\includegraphics{diagrams/grazerTraffClose.pdf}
\caption{Characteristic Traffic of a Grazer Update. Time intervals of 10ms.}
\label{fig:grazerTraffClose}
\end{figure}

Because the update took almost half a second to complete this places a restriction on the real-time capabilities of LAMS in a scenario where Grazer and LAMS are on different machines. Due to this limitation and the highly bursty nature of the traffic generated by Grazer, it is recommended that Grazer and LAMS are run on the same machine.
\vspace{4in}
\subsection{Impact of LAMS clients}

Network administrators interact with the LAMS virtual world by connecting to the LAMS server with a client. Theoretically a maximum of 32 clients can connect to a Open Arena server \cite{www:openarena} (which is what L3DGEWorld uses as a visualisation platform). We were interested in the traffic produced as multiple administrators can manage and collaborate a distribution of VoIP servers. 

To test traffic generated by LAMS client's we captured incoming traffic to LAMS server for 1 to 4 clients. Each test lasted at least three minutes and the traffic analysis data presented here is from the second minute of the test. Two common forms of gameplay were tested. First, inactive gameplay was tested to investigate the scenario where a player is connected to the LAMS server but not moving or interacting with the virtual world. Second, normal gameplay was tested where the player is moving between levels, interacting with entities and other players.

A LAMS server was ran on an Intel Celeron 2.4 GHz with a standard install of FreeBSD 7.0. LAMS and a small memory/cpu logger on the server. Traffic capturing occurred on a bridge connected directly to the LAMS server.  All traffic from all clients went through this bridge which was connected to a 100 Mbps Ethernet LAN. Figure \ref{fig:lamsclienttesting} describes our test setup.
\vspace{1in}

\begin{figure}[htp]
\centering
\includegraphics[scale=0.8]{diagrams/LAMSClientTestingSetup.pdf}
\caption{LAMS Client testing setup.}
\label{fig:lamsclienttesting}
\end{figure}

\vspace{2cm}
The instantaneous traffic received from inactive LAMS clients is shown in Figure \ref{fig:clientTraffNoMove}. The traffic rate is quite constant with only a minimal amount of short term variation visible as the number of clients increases. The average traffic rates are shown in Figure \ref{fig:clientAvgTraffNoMove}, extrapolated out to 10 clients through linear regression. The traffic received by LAMS increases linearly with the number of inactive clients. This is as expected because each client's activity (or inactivity in this scenario) is independent from the other, so there is no additional information about player interaction to send.

\begin{figure}[htp]
\centering
\includegraphics[width=5in]{diagrams/client_traff_wo_move.pdf}
\caption{Traffic generated by inactive LAMS clients. Time intervals of 1 second.}
\label{fig:clientTraffNoMove}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics[width=5in]{diagrams/client_traff_avg_wo_move.pdf}
\caption{Average traffic generated by inactive LAMS clients}
\label{fig:clientAvgTraffNoMove}
\end{figure}

When players are active in the LAMS virtual world and behaving normally the traffic rate received by LAMS is more varied, as shown in Figure \ref{fig:clientTraffwMove}. The variation increases with the number of players. For 5 players the standard deviation is 3.35 Kilobytes, significantly greater than 5 inactive clients with a standard deviation is 670 Bytes. However a linear regression model for average received traffic by LAMS is still a fairly useful approximation for small numbers of clients, as depicted in Figure \ref{fig:clientAvgTraffwMove}.
\vspace{4cm}
\begin{figure}[htp]
\centering
\includegraphics[width=5in]{diagrams/client_traff_w_move.pdf}
\caption{Traffic generated by active LAMS clients. Time intervals of 1 second.}
\label{fig:clientTraffwMove}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics[width=5in]{diagrams/client_traff_avg_w_move.pdf}
\caption{Average traffic generated by active LAMS clients}
\label{fig:clientAvgTraffwMove}
\end{figure}



\begin{figure}[htp]
\centering
\includegraphics[width=5in]{diagrams/lamsCPU.pdf}
\caption{LAMS CPU usage.}
\label{fig:lamsCPU}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics[width=5in]{diagrams/lamsMEM.pdf}
\caption{LAMS Memory usage.}
\label{fig:lamsMEM}
\end{figure}


The CPU usage increased as a LAMS client connected (Figure \ref{fig:lamsCPU}). There did not seem to be any difference in the CPU usage between a LAMS client stationary and inspecting the VoIP system. This is probably due to the short period of the testing. Memory allocated (Figure \ref{fig:lamsMEM}) to LAMS by FreeBSD did not show much variation as CPU and network traffic. 

From our results, we can see that a LAMS server can clearly be ran on a Intel 2.8GHz machine with 512 MB of RAM. 




\newpage
\section{Visual Interpretation \& User interaction}

During our LAMS client testing, we asked the opinion of several testers on our choices of visual representation. The following were the main issues with our visualisation:

\textit{Two entities for one VoIP server}: The notion of having two entities representing one machine had to be explained. It has been suggested the the server entity (pyramid) and telephony entity (asterisk) should be on top of each other to distinguish that it represents one machine with two types of statistics. Figure \ref{fig:suggestion1} describes suggestion. 

\begin{figure}[htp]
\centering
\includegraphics[ scale=0.8 ]{diagrams/suggestion1.pdf}
\caption{Stacking the server and telephony statistics to represent one machine instead of side by side.}
\label{fig:suggestion1}
\end{figure}

\textit{Unknown Visual Characteristics}: We also had to explain what each movement described in the VoIP and server system. A suggestion was made that a table or sign of some sort was placed inside the LAMS virtual world. It would give users a chance to learn quickly what the visualisation choices were. An example of a map would look like Figure \ref{fig:suggestion2} .

\begin{figure}[htp]
\centering
\includegraphics[ scale=0.7 ]{diagrams/suggestion2.pdf}
\caption{Placing signs in the LAMS virtual world to indicate to users the visual mapping characteristics.}
\label{fig:suggestion2}
\end{figure}

\textit{More monitoring and management tools}: Another minor suggestion was the choice of statistics shown in the status window and how there were limited tools to use. We only implemented two tools in L3DGEWorld. One to hang a call up and the other to inspect the statistics of a certain client. Other tools that could be implemented are tools to listen in on a call; tool to bring up the last dialled calls of a client and a tool to bring up a console. 





